Final Project

Jacob Tiede

My Adversarial GAN Setup on More Complex Data: Caltech-101

In this notebook I will test my setup for generating adversarial noise on images using a more complex dataset than MNIST: caltech-101. I had originally wanted to use ImageNet for this since it is really the standard for most of these models, but it looks like my harddrive might not be able to hold a 150GB dataset (it would fill up all of the remaining space on my harddrive, and I've heard that this can present problems when running programs). Since this seems out of my reach without some upgrades to my computer, I will opt to look at another dataset with many classes: caltech-101 (link to the dataset: http://www.vision.caltech.edu/Image_Datasets/Caltech101/). I will start by copying EDA on this data which I did in my sixth biweekly report:

Some classes are overrepresented in this data. This may be a problem, but I don't believe that over or undersampling will provide much of a benefit, since that may introduce new bias into the data (over sampling may make the neural net more sensitive to repeated examples, and undersampling will lose some potentially important data for making generalized algorithms). Now I will print some grey scale images to get an idea of the content of the data is:

There are many classes in this data, but all of them do seem to be the focus of the picture. This is a bias that may actually help our classification neural network, and since the goal is to make it make mistakes I think this bias is okay to leave in (since anything that may help our classification network will actually make the job of the generator harder). Now I'll look at a histogram of average image values:

This looks slightly skewed so we can try normalizing:

Normalization did not remove this skew, so I believe that this step is unnecessary since it would appear that the images are already approximately normal. We can now move on to the application of our neural network. Note: I will start from scratch so that I can rerun just this section of code for the purposes of testing (that way I won't need to rerun the EDA every time):

You'll notice that I opted to do some very substantial down sizing of the images this time. This is because it already took my computer a substantial amount of time to work with MNIST, so I wanted to make sure that the images were low enough resolution so that I could experiment with the model in a reasonable amount of time. This also allows us to use a much larger batch size, which is advantageous for the purpose of my proposed model's speed. We will now train the $\Phi$ neural network:

This accuracy is not great, but it is also not terrible (especially since there are so many classes). I don't think I need to really optimize the accuracy of $\Phi$ since the main point of this is to trick it, so I'll leave it as is for now. Implementing my GAN setup:

These results are actually quite amazing, and are much closer to what we expected, ie imperceptable noise that changes the output of the neural net $\Phi$. However, we can't get too excited about it since, if we look at the loss from the network applied to the normal testing images, and the ones with our generated advisarial noise added to them, we can see that they aren't drastically different. This is most likely due to the fact that we are trying to trick our neural network by minimizing the loss associated with the correct class. It looks like our neural net is never all too sure about it's answer, so our generator's job is much easier for this dataset than MNIST, because it doesn't need to push the probability down nearly as far. Still, these results are quite good especially since the generator never saw the test set of images, and was still able to find this very low magnitude noise that will almost always cause $\Phi$ to misclassify the image (when it would have normally been correct).

Final Thoughts

It would appear that, for more complex data, it is actually easier for our generator to fool $\Phi$. This is most likely due to the fact that we are minimizing the probability that $\Phi$ is correct, so for data where $\Phi$ is less sure of its classifications, it is much easier to make an unsure neural net less sure than it is for a very sure neural net to become unsure. Given more computational resources I would like to try and create a $\Phi$ that is much better at classifying than ours is, but this seems to be out of my reach for now, but it would be an interesting experiment to see if we could still get these amazing results.